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THE OBJECTIVES OF THE "TEACHING SYSTEMS PROJECT" WERE 
(1) TO PROVIDE A COMPREHENSIVE MODEL OF THE GENERALIZED 
automated teaching system, <2) TO EXPRESS THIS MODEL IN 
MATHEMATICAL TERMS AND DETERMINE THE MAGNITUDE OF THE 
CONSTRAINTS, AND (3) TO EXPLORE THE COMPUTER FUNCTIONS IN AN 

automated teaching system, experiments using various kinds of 

PROGRAMING METHODS ARE DESCRIBED. THE CONCLUSIONS FROM THESE 
EXPERIMENTS LED TO THE DEVELOPMENT OF A RATIONALE FOR A STUDY 
OF THE ADAPTIVE ABILITY OF COMPUTERS TO MEET THE LEARNING 
NEEDS OF INDIVIDUAL STUDENTS. THE CONCLUSIONS WERE-- (1) THE 
TIMES REQUIRED FOR LEARNING FROM VARIOUS METHODS OF PROGRAMED 
INSTRUCTION WERE S I GN I FI CANTLY DIFFERENT, (2) THE METHODS OF 
PRESENTING PROGRAMED SELF-INSTRUCTIONAL ITEMS WERE NOT AS 
IMPORTANT AS THE SEQUENCING OF THE ITEMS, AND (3.' BOTH LINEAR 
AND BRANCHING PROGRAMS COULD PRODUCE JUST AS MUCH LEARNING AT 
LESS COST BY USING A CARD FILE AS BY USING A COMPUTER. 

FURTHER INVESTIGATION WAS DEVOTED TO THE PROBLEM OF 
ESTABLISHING THE RATIONALE FOR INVESTIGATING THE CAPABILITY 
OF A COMPUTER TO CONTINUOUSLY ALTER ITS LOGIC PROGRAM OR 
STRATEGY DURING THE PROCESS OF STUDENT LEARNING, (AL) 
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FOREWORD 

The research described in this report, "Basic Properties of an 
Automated Teaching System", by H. W. Case and A. Roe, was conducted 
under the sponsorship of the United States Office of Education, Depart- 
ment of Health, Education and Welfare under Title VII of the National 
Defense Education Act. 

This is the final report for the Teaching Systems Project under 
Grant 7-04-138.01. 
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The objectives of the Teaching Systems Project were: 

a. To provide a comprehensive model of the generalized 
automated teaching ^ystem. 

b. To express this model in mathematical terms and 
determine the magnitude of the constraints. 

c. To explore the computer functions in an automated 
teaching system.,, 



In the early stages of the investigation, a number of experiments 
and studies were conducted, p>rimarily to acquaint the project personnel 
with the salient features of programmed instruction. 

( 10 )* 

The first experiment examined the effect of presenting to 186 
freshman engineering students course material on elementary probability 
via multiple -choice teaching machines, free -response teaching machines, 
programmed texts requiring overt responses, programmed texts requiring 
no overt responses, programmed lecturers, and standard lecturer. Though 
the time required for learning under the various methods was significantly 
different, there was no significant difference in the criterion test per- 
formance of the students using the various programmed methods of in- 
struction, and the students using programmed methods did signjlficantly 
better than those who had the standard lecture. Six months later there was 
no significant difference between the performances of the students who had 
had the different methods of instruction, . 

One of the inferences drawn from this experiment was that the 
method of presenting programmed self-instructional items was not as im- 
portant as the careful sequencing of these items. To determine whether 
this was indeed a valid assumption, 36 freshman psychology students were 
matched according to their prior mathematical ability as shown by quantitative 
scores on College Entrance Board Examinations and divided into two groups, 
one of which was given an ordered sequence of 71 items on elementary 



Numbers in parenthesis refer to publications listed in Appendix B, 
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probability, and the other was given a scrambled sequence of the same 
items. After the learning session both groups took the same criterion test. 
There was no significant difference between the groups, either in the 



learning time, error score during learning, criterion test score, or cri- 



terion test time.^^^ However, in a subsequent experiment,^^^^ the 



sequencing assumption was again examined using a longer and more difficult 
program. In this case students viewing the scrambled sequence required 
more time and scored lower than students viewing ordered sequences. 



Our early experiments made use of linear programs which required 
the student to go through every item in the program. Different presentations 
of the linear programs failed to produce significantly different terminal 
performances. However, significant differences had been obtained in the 
amount of time required to complete the learning session. It was therefore 
decided to give attention to those methods which might speed up the learning 
process without adversely affecting terminal performance. This meant 
using some kind of branching program -r- but which one? 



In preparation for experiments on the effects of different branching 
procedures, two deidces were designed. One was a random-access film 
projector with a device foi' recording the items viev/ed by the students, and 
the other was a random-access card file. The estimated cost for producing 
the film projector was in excess of $70 each; the individual card file cost 
less than one dollar. The card file was designed so that the student would 
not be aware of the particular type of branching procedure used in his course 



of instruction. It also provided a record of which items the student saw and 

( 12 ) 

his response to each item. 



Using the card file, a pilot study was conducted with branching 



methods during May, 1961, in the freshman engineering laboratory course. 

( 11 ) 



In September, 1961, a full-scale experiment' ' was conducted, with the 
various branching programs given to 189 engineering freshmen as a part '^f 
their regular instruction. The group was divided into aptitude thirds on the 
basis of their engineering entrance examination scores and assigned randomly 
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to the following seven method groups. The first group (linear) worked 
through each item. The second group (forward) branched forward if their 
responses to key items indicated a knowledge of the particular concept 
being dealt with. The third group (backward) saw each item but at the 
end of a group of items covering a particular concept, if their responses 
indicated they had not mastered that concept, branched back to the same 
sequence of items a second time. The fourth group (backward alternate) 
followed the same procedure as the third group but instead of branching 
back to a repetition of the same items, they were branched to alternate 
items covering the same concept. The fifth group (pre-test forward) dif- 
fered from the forward-branching group in that they took the criterion test 
before they had a chance to see the program and then repeated the test at 
the end of the program. The sixth group (random) worked through the 
same items as the linear group, but the ordering of items was random 
rather than sequential. The seventh group (text) was given the forwar.d 
branching program in the form of a textbook instead of the card file. 

In all the branching programs, two key branching items were used 
ir, preference to a single one to test the student’s mastery of a concept, m 
order to reduce the possibility of the student's guessing the correct answer 
and thereby skipping over a sequence of items which he did not really know. 

In evaluating the results of the experiment, the first question posed 
was whether the program resulted in any significant learning at all. There 
was indeed a very significant increase in the mean test scores from pre- 
test to post -test (29. 3 to 63. 8, significant at < . 0005). Similarly there was 
a significant reduction in the time required to complete the test items 
(39.6 to 21. 7 minutes, significant at <. 0005). 

The various branching methods were then compared. No significant 
interactions were observed between aptitude thirds and methods. Hence, 
results were treated by covariance design to remove the effect of aptitude 
on learning methods. An overall analysis of variance for methods showed 
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a significant difference between the groups in test scores (at , 01 level) and 
in learning times (at . 0005 level). But no significant difference in test 
I scores was observable between the "backward" and "alternate backward" 

i or between the "linear" and "forward" or between "linear" and "backward" 

X 

i groups in test score. 







With regard to learning time, significant differences were observed 
between the linear and the backward groups (118- 1 and 135. 4). The dif- 
ference between the linear and forward groups was not significant (118.1 
and 110.2). As expected from the above, the differences between the for- 
ward and the backward groups was significant (110.2 and 135.2)., 



The card file was used during the experiment primarily because we 
wished to control the browsing behavior of the students and thereby get a 
valid learning time measurement. In order to find out what effect on 
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learning time and post-test performance was introduced by the card file 
arrangement, we gave one group of students a forward branching program 
in the form of a scrambled text which instructed the .tudent to follow a 
specified item sequence but also gave him the opportunity to browse if he so 
wished. The results show no significant difference between the two groups 
in test scores and in test times. But in learning time, the mean difference 
of 30 minutes was very significant (at <. 0005 level) in favor of the texi group. 
It is very likely that the students browsed through the text, which would 
account for the learning time difference. 



i 



Subjective opinions of students about the various branching methods, 
indicated by their liking ratings, did not correlate either with their aptitudes 
or with their performances on the criterion tests. 



One of the conclusions that emerged from this experiment was that 

learning time was one of the more important variables, one which could be 

significantly affected by differences in the teaching procedure. However, 

we became increasingly aware of the multitude of contingent circumstances^ 

(15) 

which affect the teaching-learning process, and became convinced that 
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an efficient data handling and logic device would be required. '* We had 
observed, however, that digital computers were being used to select items 
to be presented to students based on the student’s responses to previous 
items and on a preconceived, fixed set of branching rules. Also, the com- 
puters were being used to gather and process data on student performance 
for period ic review by the experimenter or teacher. Our experience with 
the simple card file indicated that these particular functions could be ac- 
complished in a far less expensive fashion than by using an "on-line" com- 
puter in the teaching system. The particular computer function which wc 
were interested in was the adaptive capability of the computer to continuously 
alter its logic program or strategy during the process of student learning. 



The usual approach of specifying a fixed strategy for controlling 
the teaching stimuli, based on a particular principle of learning, is a non- 
optimizing approach and is only as good as the choice of the particular 
principle of learning underlying the strategy, or the skill with which the 
principle is translated into a strategy (or computer program). Our approach 
is to attempt to improve student performance by continuously varying and 
comparing possible strategies. In such a system, students are both learners 
and "experimental subjects", and the traditional experimental approach of 
ignoring the effects on subjects who have been exposed to suboptimal regimens 
cannot be tolerated. 



We were further convinced that a prerequisite for the logical design of 
a teaching system was to have some measure of student performance which 
reflected a value or "utility" outside of the system. Without such a value on 
the output of a teaching system it would be difficult to justify allocation of 
resources in the system. 



The "systems" we considered could be roughly divided into four 
categories. 



Micro-micro systems: Concerned with a transformation 
of students’ behavior by a single, relatively short sequence 
of learning items. 
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Micro systems}; concerned with a transformation of 
students' behavior by a longer sequence of learning 
items, such as are encountered in a semester course. 

Macro systems; a collection of micro systems characterized 
by a curriculum or curricula in a school, university or sc>’''ol 
district. 

Macro-macro systems; related to the transformation of 
students' behavior by the total learning experience 
encountered during the students' lives. 

Educational systems could be further classified (regardless of size) 
according to the extent of adaptivity, as follows; 

Zero Level Adaptive Behavior : A fixed, preconceived strategy 
(or pedagogy) is used for presenting to all students a fixed, 
preconceived set of courses or list of subject matter. 

First Level Adaptive Behavior ; A fixed strategy which uses an 
individual student's past history of performance to determine 
which particular course or list of subject matter from a precon- 
ceived set of such courses or subject matter is shown to that 
individual student. 

Second Level Adaptive Behavior; The particular courses or list 
of subject matter which is shown to a particular student is 
determined by a fixed strategy which uses an individual student's 
past history of performance and the history of performance of 
all students who have previously gone through the system. 

Third Level Adaptive Behavior: A set of strategies for presenting 
students with courses or lists of subject matter is available. The 
choice of a particular strategy for a particular student depends 
on the history of performance for each of the strategies. 

Our interest centered on providing a rationale for designing an auto- 
mated teaching system exhibiting third level adaptive behavior. The elements 
of this rationale would consist of; 
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a. Provisions for data gathering and processing 

b. A criterion function 

c. Decision rules 



d. A utility function 



Each of these elements are described in Report No. 63-63 in some detail 



and only brief mention will be made here of a,b, and c. 

a. Data gathering and processing . The assumption is made that the 
amount of data that is required for an adaptive automated teaching system is 
sufficiently voluminous to require the use of modern data processing equip- 
ment. Such equipment may or may not be used for presenting course content 
material directly to students (as in the computer -controlled teaching 
machines). Another assumption, somewhat contingent on the use of modern 
data-processing equipment, is that flexible scheduling (for individual students, 
at any time during the school year) will be employed. 



b. A criterio n function is developed for an adaptive educational 
system where two processes are being carried out simultaneously, namely. 



(1) students are learning subject matter, and (2) the system controllers are 
learning about the student's learning. Process (2) may include exploratory 
use of various alternative pedagogical procedures or subject matter, some 
of which may result in better student performance ^han others. In such a 
situation there is a trade-off between processes (1) and (2). The suggested 
criterion function is the sum of the net utility of all students' outputs, and 
obviously this function should be maximized. 

c. The decii 
function will depend 
probability distribut 
63-63 contains decis 
important being for 
assign individual sti 
the sum of their net 
bandit" problem, ar 








unknown means had not been previously found. Report No. 63-63 presents 
a computational backwards -induction solution to this problem, and one 
which is of particular interest to computer -mediated teaching systems, 
since the proper action to take for each possible contingency can be 
determined before the system goes into operation, and stored in memory. 

An example of the solution for assigning students to one of two learning 
situations is illustrated in Figure 1. 

d. The utility function is used to provide a measure of student per- 
formance, or output. It converts such available measures as student grades, 
student learning time, teacher inputs, school capital and maintenance costs, 
etc. , into a net value of the transformation effected by the system. To ob- 
tain the utility, one first projects the future expected life cycle earnings of 
the average student going through a particular educational experience from 
the history of life earnings of students who have previously gone through a 
nominally similar educational experience. (See Figure 2 for example of 
such a projection for the engineering student graduating in 1962. ) In order 
to use this average expected life cycle earnings for a specific student, some 
adjustment must be made for individual differences. The most widely 
available measures of individual student differences are class standing 
and scholastic grades. Interestingly enough, many studies on the relation- 
ship between school performance and subsequent earnings fail to indicate 
a correlation. This apparently is true for the first ten years after a student 
leaves school (and many of the studies which report no correlation do not 
trace a student's career beyond the 10 year mark). However, for technically 
trained people at least, earnings tend to diverge ten years after graduation. 
D.S. Bridgman, in "Success in College and Business", Personnel Journal , 
Vol. 9, 1930, pp. 1-19, quotes Gifford's 1928 study of Bell Telephone System 
college graduate employees, where higher salaries were associated with 
higher class standing and vice versa (see Figure 3). Somewhat similar 
findings were obtained in a more recent (1962) study of over 10, 000 Bell 
Telephone System college graduates, and in an analysis made by our group 
on data obtained on engineering graduates from tl. University of California. 
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The critical Aj, surface is shown for k = 2, a^ = a^, N = 58, 
To use the figure locate the grid point corresponding to the 
number of observations, n®’, previously taken from the 
category with the larger current sample mean and 
the nxunber of observations, n^, from the 
category with tne sm.aller current sample 



mean. At this grid point, determine Aq 
by interpolation between the contour lines 
If the observed Ag (defined below) is 



less than A^,, select the next 



observation from the 
category with the 
smaller sample 
mean. 




If the 

observed Ag 
is equal to or larger 
than Ap, select the next 



observation from the category 
with the larger sample mean. 



Observed A = Larger sample mean ■ Smaller sample mean 
E Standard Deviation of Category with larger 

sample mean 



DECISION SURFACE 
FIGURE 1 
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From this recent data, approximate correction factors are obtained as 
shown in Figure 4. Applying these correction factors to the average ex- 
pected life cycle earnings , one obtains a set of curves similar to the upper 
lines on Figure 5. The lower lines represent the expected life cycle 
earnings or persons who have not gone through the specified educational 







EFFECT ON PRODUCTIVE OUTPUT FROM INDIVIDUAL 
AND EDUCATIONAL DIFFERENCES 

FIGURE 5 

i 

The next step is to select a discount rate and find the present worth 
i of the expected life cycle earnings of the individual student. From this is 

subtracted the present worth of the expected life cycle earnings oi matched 
I individuals who have not gone through the educational experience. The re- 

suit is the "utility" or net output associated with a given educational 
fi experience. 



Discounting future earnings, aside from being a common economics 
practice, has two important features. First, it reduces the error in the 
estimate of the utility by giving less weight (e.g. , discounting over a longer 
period) to earnings occurring later in the life cycle. Secondly, it solves 
the problem of how to evaluate the trade-off between the increment in student 
performance and the increment of time necessary to obtain such an in- 
crement of performance. An example borrowed from Report No. 63-63^^^^ 
will illustrate this point. If the average engineering student could complete 
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his college studies in three years instead of the customary four years, 
the increment in the present worth of his expected life cycle earnings 
would be $12, 600. Report No. 63-63 goes on to indicate that such an 
engineering student could theoretically afford to spend up to $6, 800 for 
each of the three years in an accelerated program, as compared to approxi- 
mately $1, 800 for each of the years in the normal four-year program. 




Another variation of this problem is to calculate the additional 
amount of resources one would be willing to commit to education if these 
additional expenditures resulted in a student getting an M.S. Instead of a 
B.S, degree in four years. 



Somewhat more speculative, since it introduces the additional un- 
certainties of the relationship between school performance and subsequent 
professional performance, is the problem of calculating the additional 
amount of resources one would be v/illing to commit to education if these 
expenditures resulted in a student getting, say, an A average instead of a 

B) average. 

It is relatively easy to specify a utility for those macro systems 
where sufficient historical data exist for making projections. The problem 
is more difficult for micro and micro-micro systems, and the present 
approach, pending the accumulation of further data, is to make a number 
of simplifying assumptions. Many of these assumptions are discussed in 
detail in Report No, 63-63. 
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APPENDIX A j 

PERSONNEL ASSOCIATED WITH THE TEACHING SYSTEMS PROJECT j 

Dr. Harry W. Case, principal investigator, 1961-1964 ^ 

Dr. Arnold Roe, co-principal investigator, project director, 1961-1964 i 

Dr. Mildred Massey, statistician, 1961 
Dr. Henry Kramer, mathematician, 1962-1963 

Dr. Yasuko Filby, research psychologist, 1961 I 

K. M. Srinivas, research psychologist, 1961-1963 j 

William Kirk, Jr., research psychologist, 1962 j 

Richard Nazro, computer and instructional programmer, 1961 | 

Steven Crocker, computer programmer, 1961 j 

Jane Osuga, computer programmer, 1962 | 

Glen Johnson, computer programmer, 1963-1964 \ 

Richard Pierce, research engineer, 1961 I 

Arieh Lewin, research engineer, 1961 j 

Stephen Jensen, research engineer, 1962 

Paul Englund, research engineer, 1963-1964 I 

Dora Stein, technical writer and office administrator, 1961-1962 j 

Phyllis Brunner, clerk-typist, 1961-1962 | 

Nancy Goldin, clerk-typist, 1963-1964 j 

Penelope Mitchell, clerk-typist, 1963-1964 | 

Jeanette Griver, student, 1961 | 

Mark Sink, student, 1961 

Rollie Winters, student, 1961 I 

Vassiliki Vlachouli, student, 1961 j 

Ronald Baldwin, student, 1961 j 

Carl Pignoll, student, 1962 I 
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The automated learning research program in the Department of 
Engineering. Arnold Roe, UCLA Alumni Mag . , 1961, 3 ^ No. 5, 
p. 14-16. 

Branching programs in automated instruction: a simplified format. 
Arnold Roe, University of California, Los Angeles, Department of 
Engineering, Report No, 61-72 , August 1961. 

A dynamic statistical approach to the control of human behavior. 
Arnold Roe and John Lyman, Paper presented at APA, New York, 
September, 1961. 

An evaluation of auto -instructional techniques. Harry W. Case and 
Arnold Roe, Paper presented at APA, New York, September 1961. 
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The use of linear and branching programs in freshman engineering 
courses. Harry W. Case and Arnold Roe, Paper presented at Pacific 
SW Section, ASEE, Las Vegas, December 1961. 

Learning to operate a small computer by auto -instructional text. 

Arie Lewin, Arnold Roe and Thomas J. Woodson, Paper presented 
at Pacific SW Section, ASEE, Las Vegas, December 1961. 
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7. Research on teaching machines and programmed learning. Arnold 
Roe, J. Engr. Ed. , 1962, 5^, pp. 439-446. 

8. Scrambled versus ordered sequence in auto -instructional programs. 

K. Vlachouli Roe, Harry W. Case and Arnold Roe, J. Ed, Psychol. , 
1962, No. 2, pp. 101-104. 

9. Programmed learning experiments in engineering at UCLA. Arnold 
Roe and Harry W. Case, Paper DP 62-640 presented at AIEE meeting, 
St. Louis, April-May 1962. 

10. Automated teaching methods using linear programs. Arnold Roe, 

J. Appl. Psychol ., 1962, ^ No. 3, pp. 198-201. 

11. A comparison of branching methods for programmed learning. 

Arnold Roe, J. Ed. Res. , 1962, pp. 407-416. 

12. Format for branching programs in automated instruction. Arnold Roe, 
IRE Trans, on Ed. , 1962, E-5, pp. 131-135. 
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16. 
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A report on programmed instruction. J. Engr. Ed., 1962. 53. 
pp. 117-123. — 



The dynamics of an automated teaching system. Arnold Roe, 

John Lyman and Harold Moon. In Applied programmed instruction. 
Stuart Margulies and Lewis D. Eigen, eds., John Wiley, New York. 
1962. 



Research in programmed learning. Arnold Roe, In Programmed 
learning and compute r-based instruction. John E. Coulson. ed. . 
John Wiley, New York, 1962. 



Programmed learning in freshman engineering classes. Arnold Roe, 
Mildred Massey, Gershon Weltman and David Leeds. In New media 
in higher education , James W. Brown and James W. Thornton, Jr. , 
eds. Assoc, for Higher Educ. and NEA, Washington, D.C., 1963. 



An adaptive decision structure for educational systems. Arnold Roe, 
University of California, Los Angeles, Department of Engineering, 
Report No. 63-63, December 1963. 



Preliminary draft for the panel on teaching methods. Arnold Roe, 
In Honors programs in engineering, Frank Kreith and Jeremiah 
M. Allen, eds., Allyn and Bacon, Boston, 1964. 
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